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ABSTRACT 

Start, a family literacy program in Lincoln (Nebraska), and an 
evaluator collaborated to create a low-stakes, small-scale portfolio 
assessment to collect information on program impact. The small scale 
of the evaluation offered opportunities to involve the staff in each 
step of the portfolio creation process. Lincoln*s Even Start program 
consists of four components: adult basic education, family literacy, 
early childhood services, and home visits. Family literacy portfolios 
were planned to demonstrate changes in parent-child interactions and 
reading resulting from the programs and home visits. Portfolios were 
prepared for 33 families, 17 of which spoke a language other than 
English as their primary language. Scoring rubrics were developed and 
participant results from the Home Screening Questionnaire (C. Coons 
and others, 1981) were used as an indicator of the validity of 
portfolio scores. The assessment effort showed that it is possible to 
create a portfolio system that measures family growth in literacy and 
parent-child relationships. Implications for program evaluation and 
continued portfolio development are discussed. An appendix contains 
checklists used in portfolio development and the portfolio scoring 
rubric. (Contains 2 figures , 7 tables , and 11 references .) (SLD) 
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INTRODUCTION 

In How Shall We Study Comprehensive, Collaborative Services for Children and 
Families?, Knapp (1995) noted that the trend towards integration of education and 
human services poses challenges in evaluciiing program impact. Since collaborating 
agencies often operate under differing assumptions, the evaluator must fashion an 
evaluative framework that integrates divergent views. Knapp posited that the design of 
the evaluation could benefit from collaborative efforts of the evaluator, service 
recipients, program staff, and researchers associated with each discipline. 

O’Sullivan (1995) reported that agencies are becoming more proactive in 
determining the evaluative framework of their programs. She noted that the evaluator’s 
role has become increasingly collaborative as clients seek evaluative information from 
assessments that are more closely aligned to agency missions and goals. In addition, 
agencies rely on technical support from evaluators in the implementation and review of 
assessments. Finally, clients collaborate with evaluators to interpret assessment 
results and make program decisions. 

The role of the evaluator has become that of a collaborator who can assist 
program staff in designing an assessment system that is aligned with the vision of the 
agency and that offers quality information for making decisions. The development of 
alternative assessments is technically demanding. Specifically, programs interested in 
a more individualized, holistic assessment are faced with the challenge of increased 
teacher workloads and the unreliability of scoring (Koretz, McCaffrey, Klein, Bell & 
Stecher, 1992). For example, reliability for scoring of portfolios in the Vermont 
statewide assessment ranged from .33 to .43. 

The scoring of assessments -- once solely the province of the “outside, unbiased 
judge” -- has become increasingly collaborative as staff members become involved in 
scoring. In their work with a small collection of student writing portfolios. Moss and 
others (1992) also reported low rater reliabilities. To improve the quality of the ratings 
the authors proposed that the scoring of portfolios should involve those most intimately 
Involved with the program. The state of Kentucky has Involved teachers in the scoring 
of students’ portfolios (Advanced Systems in Measurement and Evaluation, 1993). 
Classroom teachers scored students’ portfolios and an outside agency validated the 
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scoring by rescoring a random sample of portfolios. Thus, collaboration is increasingly 
playing a role in the entire evaluative/assessment process: specification of 
goals/objectives, development of assessments, scoring of assessments, interpretation 
of results, and subsequent planning. 

The use of a portfolio system of assessment for program evaluation presents the 
opportunity for evaluators to collaborate with clients to identify program goals, and align 
service delivery with those goals. In addition, a small-scale evaluation presents the 
opportunity for the evaluator to collaborate with staff members to develop assessments 
aligned with program goals and to share the research base necessary to avoid pitfalls 
associated with the use of alternative assessments. This interactive development and 
implementation of an assessment has the potential for greater program impact. 

During the 1994-1995 program year, the staff of a family literacy program and an 
evaluator collaborated to create a low-stakes, small-scale portfolio assessment to 
collect information on program impact. The scale of the evaluation offered 
opportunities to involve the staff -- program coordinator and family educators - in each 
step of the creation of the portfolio system. The purpose of this study is to describe the 
collaborative process and the psychometric quality of the assessment. 

The Program 

Even Start is a federally-funded, family literacy program. The 1988 Hawkins- 
Stafford Amendment provided the federal initiative for Even Start, and some of the first 
projects were funded in 1989. To be eligible for Even Start services the participant must 
be an adult/young adult in need of literacy skills who is responsible for a child age 
seven or younger.^ Even Start programs provide three types of core services; adult 
education, parent education/child development services, and early childhood sen/ices 
(St. Pierre, Swartz, Murray, Deck, & Nickel, 1993). The program is mandated to be 
collaborative in the delivery of services and functions to coordinate sen/ices among 
local, state, and federal agencies to meet the needs of clients so they may pursue a 
literacy program. As part of these services, the program provides such support services 



' Hereafter the term parent will be used to describe the person who serves as the child's parent or legal 
guardian. 
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as transportation and developmental childcare while parents attend classes to improve 
their literacy skills. 

The Lincoln Even Start program began during the 1992-1993 school year and 
completed its first evaluation during the 1994-1995 program year. During the 1994- 
1995 program year, the Even Start program of Lincoln, Nebraska provided continuous 
services to forty-seven families. Twenty-six of the continuing families speak English as 
their primary language. Twenty-one of the families speak either Vietnamese, Russian, 
or A: able as their primary language and English as a second language (ESL). 

Lincoln's Even Start Program consists of four core programs: Adult Basic 
Education (ABE), Family Literacy, Early Childhood, and Home Visits. Parents enrolled 
in Even Start must make a commitment to participate in all four areas. A brief 
description of each component follows: 

Adult Basic Education: Meeting the literacy goals of the adult is the fobus of 
this core service. The adult participating in Even Start has a variety of options to 
pursue in terms of her/his literacy goals. For example, to achieve her/his literacy 
goals, the parent may attend a General Educational Development (GED) class or 
ESL class, meet with a tutor, or attend an employability program. These 
services are provided by collaborating agencies in the Lincoln community. 

Family Literacy: In this component, the parent attends a class on how to 
support his/her child's development and education. As part of this component, 
the parent and child participate in a planned activity termed Parent And Child 
Together (PACT) that applies the ideas learned in the class to the parent's 
interaction with the child. These services are provided by family educators who 
are part of the Even Start staff. 

Early Childhood: The emphasis of this core area is on meeting the child’s 
developmental needs. During the parent's attendance at literacy classes, the 
child participates in a developmentally appropriate early childhood program. 
These service are provided by collaborating agencies in the Lincoln community. 
Examples of current providers of child care include licensed developmental day 
cares, pre-schools. Head Start programs, and Lincoln Public Schools. 
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Home Visits: In this component, the focus is on the parent and child in the 

home setting. Home visits are made by the family educator, and activities 

consist of adult basic education, family literacy, and/or PACT. 

The Evaluation 

At the national level Even Start has an information .system which collects 
information on local programs in four areas: 

1 . Characteristics of local Even Start projects, including the types of projects 
that have been funded, the services that they provide, the collaborative 
efforts they have undertaken, and the obstacles that exist to program 
implementation; 

2. Characteristics of families who participate in Even Start; 

3. The type and extent of services that participating families receive; 

4. The outcomes of Even Start program participation on families, parent, and 
children. 

This information is used in the national evaluation of Even Start programs. All 
local Even Start programs are required to collect information in the first three areas for 
the evaluation. Local programs selected for the Sample Study are required to collect 
information about the fourth area; the Lincoln Even Start program was not included in 
the Sample Study, in addition to collection of the information for the national 
evaluation, the local Even Start staff and a member of the evaluation team identified the 
following evaluative questions for the 1994-1995 Even Start program: 

Program description: How do program services compare to services offered 
the previous year and How do program services compare to programs 
nationally? 

Adult Basic Education: As a result of participation in the Even Start program, 
what changes have occurred in the parent's life skills? 

Family Literacy: As a result of participation in the family literacy classes and 
home visits, what changes have occurred in parent/child interactions? 
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Early Childhood: Are child care agencies making progress toward providing 
more deveiopmentally appropriate services? and What changes have 
occurred in the child as a result of participating in the Even Start program? 

Program satisfaction: What factors contribute to a family's continued 

participation in the program? and What factors contribute to a family's 
decision to discontinue participation in the program? 

In the area of family literacy the director and program coordinator wanted to 
create an assessment that measured the degree and quality of parent and child 
interaction. The decision to use portfolios for this purpose was based on the alignment 
of the assessment with a holistic view of family literacy. It was also discussed that the 
development of a portfolio system could serve to inform other authentic assessment 
issues that were developing in Lincoln Public Schools. 

For purposes of validation, information about the level of family literacy was also 
collected using the Home Screening Questionnaire (HSQ), a standardized instrument 
with questions about factors in the home that affect a child’s development (Coons, Gay, 
Fandal, Ker, & Frankenburg, 1981). The Home Screening Questionnaire was selected 
for the national Sample Study to assess change in the core area of parent 
education/child development sen/icea: thus, the Lincoln administration of the HSQ will 
also allow comparisons with national results. In addition, at the end of the program year 
the family educators were surveyed to determine the impact of the program on 
activities. 

METHOD 

Development of the portfolio system began in December of the 1994-1995 
program year. The first meeting was a half-day session to familiarize the six family 
educators, the program coordinator, and the evaluator with the use of portfolios as an 
assessment tool in family literacy. The session was led by two team leaders from the 
Head Start program who were using alternative assessments in their program. The 
session began with a general overview of assessment and moved to the use of 
portfolios in assessing students. Miesel’s portfolio system, the Work Sampling System 
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(1993), provided an example for review. The application of portfolios to family literacy 
was then developed using Popp’s Family Portfolios: Documenting Change in Parent- 
Child Relationships (1992). The article provided information about portfolios and posed 
questions that the Lincoln team would need to answer prior to implementation of its 
portfolio system. * 

A week later the team met in an all-day session to review the mission of Even 
Start, to establish goals related to family literacy, and to identify types of artifacts that 
were appropriate for the portfolio. The evaluator and program coordinator of the Even 
Start program led the meeting. Also contributing to this session were the Director of 
Federal Programs, the Administrative Assistant to the Director, and another evaluation 
specialist. A general review of portfolios covered the need for core items, selection of 
evidence to reflect typical performance, participant reflection on the artifacts, letters to 
the reviewers, and issues of reliability and validity. The group then moved into a 
discussion to resolve ten questions posed in Family Portfolios: Documenting Change in 
Parent-Child Relationships (Popp, 1992, pp. 4-18). The questions and their solutions 
are shown in Table 1. 

The discussion of the purpose of the portfolio began with a review of the mission 
statement of Even Start. The mission statement of the Even Start Program in Lincoln is: 
Through the collaboration and cooperation of community agencies, the Lincoln 
Even Start Program provides a family centered program with primary goals to: 

(1) assist parents to be the primary and most significant teachers of their 
children; (2) assist parents to provide a family literacy environment for their 
children; (3) assist parents to achieve their goals related to literacy and 
education; (4) teach and assist parents in locating sen/ices for the health, 
nutrition, safety and well being of the family; and (5) assure quality education for 
children. The mission of Even Start is to empower families to transfer the 
value of education to their children and demonstrate self-sufficiency in 
their complex lives (Even Start of Lincoln, Nebraska). 




o 



Page 1 



Table 1 

Issues Addressed in the implementation . 
of the Family Literacy Portfolio 

What is the purpose of the portfolio? 

Provide evidence of family literacy. 

What physical form does the portfolio take? 

For Lincoln, the portfolio is kept in a large plastic container. 

What contents should be included in the portfolio? 

Artifacts that provide evidence of family literacy as it relates to the Even Start goals. 

The variety of artifacts is flexible; they, may include videos, audiotapes, pictures, and 
essays. Artifacts should be accompanied by written reflections about the artifact. The 
pre- and post-assessment using the Family Literacy Checklist should be in all portfolios. 

How often should families add new materials to the portfolio? 

Families should add new artifacts every two weeks. These artifacts should be reviewed 
on occasion and replaced with new ones. At the end of the year, each goal should 
have three to four artifacts that provide information about typical family performance for 
that goal. 

Who makes the decision of what goes into the portfolio? 

The parent, child, and family educator should collaborate to determine what goes into 
the portfolio. 

Who owns the portfolio? 

Until the end of the program year, the contents are owned by the family and the Even 
Start program. 

What happens to the portfolio at the end of the year? 

The family owns the contents. 

Who ha 5 access to the portfolio? 

The family and Even Start staff. Cooperating agencies would need to obtain written 
permission for review of the portfolio. 

Who will assess/evaluate the portfolio contents? 

Even Start staff and the evaluator. 

When do portfolios need to be completed? 

The first week of May. 
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In addition the goals and objectives vf the Even Start Family Literacy component 
were reviewed. The goals are: 

1) Build self-esteem and develop feeling of empowerment among parents and 

children. 

2) Build on a parent’s life skill ability and accomplish personal and family goal 

setting. 

3) Build on a parent’s parenting ability and help parent learn to model positive 

parenting skills for his/her children. 

4) Increase parent’s appreciation of learning and the value of education and the 

effect they have on her/his children’s learning progress. 

5) Establish and nurture feelings of connectedness and appreciation between 

the parent and child (Even Start of Lincoln, Nebraska) . 

After reviewing the mission statement, goals, and objectives, the discussion 
turned to focusing the assessment in terms of selection of goals that are the core of 
family literacy. The family educators, program administrators, and the evaluator 
reached consensus in the identification of two encompassing areas which would be the 
focus of the portfolio assessment (see Table 2). Listed under each broad area were 
the family literacy skills that parents should know and practice. 

In the afternoon session the team broke into groups to identify the types of 
artifacts which would be appropriate for inclusion in the portfolios. The formats of the 
artifacts proposed were videotapes, audiotapes, pictures, essays, and checklists. 
Tension existed between team members about the need for a standard set of core 
items for comparative purposes in scoring and the need for complete freedom of choice 
of artifacts for personalization of the portfolios. The issue was resolved by leaving.the 
format of the artifacts open; however, the artifacts were to address the family literacy 
goals as identified by the group. The exception to the open format of the artifacts was 
in the area of the Family Developmental Needs. Due to the large number of objectives 
associated with this goal, and the importance in attaining the objectives, the team 
decided that the objectives associated with family developmental needs should be used 
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to create a checklist. The checklist would provide a pre- and poet-assessment and be 
included in every portfolio (see checklist in Appendix). 





Table 2 

Areas of Focus for the Family Literacy Portfolio 


Literacy: 


LIFE SKILLS: Parent models essential life skills for child. 

Parent models reading/storytelling, math, and writing skills for her/his 
child. 


f 

Strategy 

Development: 


Parent models for his/her child goal-setting and planning of activities. 


Self-esteem: 


Parent promotes self-esteem in child by providing, for example, 
physical contact and/or positive verbal comments. 


Parents as 
Teachers: 


PARENTING: Parent uses developmentally appropriate 
parenting practices with child. 

Parent functions as her/his child's teacher. 


Parent/Child 

interactions: 


Parent and child interact in child-centered activities selected by the 
child. 


Family 

Developmental 

Needs: 


Parent creates an environment which contributes to the physical, 
social, and emotional well-being of her/his child. 



In February the team met to share the development of the family literacy 
portfolios. The meeting was led by the program coordinator and the evaluator. The 
session began with a review of the evaluation question for which the portfolio 
assessment was designed: “As a result of participation in the family literacy classes and 
home visits, what changes have occurred in parent/child interactions?” Subsequently 
the team reviewed the list of goals and potential artifacts. Artifacts collected for the 
portfolios were shared by the family educators and resulted in a discussion about the 
need for alignment of the artifacts with the goals. 



A meeting for review of portfolio collection and the development of scoring 
rubrics was scheduled for March. The session began with a review of portfolio 
collections and mapping the artifacts back to the family literacy goals. As a result of 
this meeting the team designed a portfolio log (see Appendix) to record the artifacts 
and the goal(s) for which the artifact should be reviewed. The log served two purposes: 
to inform the family educator about progress in each area and to inform the reviewer 
which artifacts may provide evidence for level of proficiency for a specific goal. 

In the March meeting the family educators, program coordinator, and 
adnrinistrative assistant provided feedback on drafts of scoring rubrics prepared by the . 
evaluator. An analytic rubric and a holistic rubric were developed using the family 
literacy goals that were the focus of the portfolios (see rubrics in Appendix). In the 
development of the analytic rubric, the goals were listed and descriptors written for the 
extremes and midpoint of a five-point scale. The descriptors for the analytic rubric were 
used to develop the narrative for the four levels of proficiencies -- Proficient, 

Developing, Emerging, Not Yet - of the holistic rubric. Narrative was added to the 
holistic rubric that developed a contrast between levels. For example, in the description 
of the Developing classification the narrative reads: ‘The predominant characteristic 
that distinguishes a Developing Level from a Proficient Level in family literacy is 
evidence of some of the life skills and parenting skills but not all - or most - of the 
skills.” The classifications represent levels of development of family literacy that range 
from no evidence of family literacy skills {Not Ye(^, to uneven development of those 
skills {Emerging), to development of many family literacy skills {Developing), to the 
presence of most of those skills {Proficient). A complete description of each level is 
included in the appendix . 

The team first addressed which rubric - the analytic rubric or the holistic rubric ~ 
was appropriate for assessing the portfolios. Some members of the team indicated that 
the holistic rubric fit philosophically with the idea of family literacy. Other team 
members indicated the analytic rubric provided more information that could be used for 
program improvement. The issue was resolved when the team reached consensus by 
deciding to use a two-stage scoring system. The analytic rubric would be scored first 



and be used to inform the reviewers in arriving at a final classification on the holis.ic 
rubric. 

The team then turned its attention to the refinement of the rubrics. Suggestions 
inciuded the refinement of language and provision of consistent ianguage across 
descriptions. A finai area for refinement invoived estabiishing more distinction between 
ievels of proficiencies. Suggestions were incorporated into the finai versions of the 
rubrics. 

At the end of the 1994-1995 program year, the team met for two days to review 
the rubrics, to initiaily score some portfoiios as a group, and finaily to score portfolios in 
pairs. Compiete portfoiios were avaiiabie for 35 famiiies. Seventeen of the families 
spoke Vietnamese, Russian, or Arabic as their primary ianguage; and 18 of the famiiies 
spoke Engiish as their primary ianguage. 

The decision was made, based on the research of Moss and others (1992) and 
the scoring practices in Kentucky, that family educators, the program coordinator, and 
the evaiuator wouid score portfoiios to provide more program feedback to Even Start 
stakeholders. Earlier discussion of having parents participate in the scoring identified 
the probiem of invasion of privacy. To reduce bias, famiiy educators did not score the 
portfolios of their families; and famiiy educators who served ESL famiiies did not score 
ESL portfoiios. 

In preparation for scoring, in a haif-day session the team reviewed the rubrics 
and reviewed portfoiios that would provide benchmarks for the ievols of iiteracy. During 
the training each member of the team scored the portfoiio individuaiiy, shared her/his 
score with the group, and then discussed her/his ratings. In the training session, the 
team agreed that one portfoiio was not scorable, two of the portfolios were at the 
Emerging level, one was at the Developing level, and one was Proficient. A final 
portfolio was scored and the ratings were all within one ievel of agreement. 

In the remaining one and one-half days, the portfolios were scored using the two- 
stage scoring. Scoring was independent untii the assessors had arrived at a finai 
classification based on the holistic rubric. Once a decision was made the assessors 
compared the holistic score. When two raters were in exact agreement the 
classification they assigned was reported. In the case of raters disagreeing one ievel. 
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the intermediate classification was assigned (e.g., Developing to Proficient). If the 
holistic score differed by more than one level of classification, the portfoltio was 
rescored by two new assessors. With one exception, score resolution for the analytic 
rubric followed the same procedures as the holistic rubric. In the case of the analytic 
rubrics when raters disagreed by more than one level, they discussed their judgments 
and negotiated the final score. 

The Home Screening Questionnaire (HSQ) was selected for the national 
evaluation as an indicator of change in family literacy. The local program decided to 
use the HSQ for purposes of comparing the results for the Lincoln Even Start program 
with results for the nation. In addition, the HSQ would provide some Indication of the 
validity of the portfolio assessment. 



RESULTS 

Consistency of Scoring 

Consistency of scoring for the holistic rubrics was 93% when using the criteria 
that scores should be within one level of agreement (see Table 3). In the scoring 
session only two of the portfolios were assigned ratings that differed more than one 
level. When the two portfolios were rescored by new reviewers, all portfolios were with 
one-level of agreement, and 63% (19) of the ratings were in exact agreement. The 
inter-rater reliability between raters’ scores on the holistic rubric was r =.66, and the 
inter-rater reliability of item-level scores ranged from r=.53 to r=.76. 

Validity Issues 

The results of the scoring are shown in Table 4. Approximately one fourth (25.7%) of 
the portfolios were classified as Developing to Proficient or Proficient. Nearly half 
(42.9%) of the participants were classified as Emerging and Emerging to Developing. 
None of tht) families were classified in the Not Yet group, and only one family was 
classified as Not Yet to Emerging. In written and verbal feedback, the family educators 
indicated that the classifications that were assigned to families with whom they worked 
were appropriate assessments. 
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Table 3 



Consistency of Portfolio Scoring 





Within One 
level of 
Agreement 


Exact 

Agreement 


Inter-rater 

Reliability 


Scores for Holistic Rubric 


93% 


63% 


.66 


Scores for Analytic Rubric 


. 






Parent models reading/storytelling, writing, math skills 
for her/his child. 


97% 


53% 


.76 


Parent models for his/her child goal-setting and 
pianning of activities. 


93% 


57% 


.74 


Parent promotes self-esteem in child by providing, for 
example, physical contact and/or positive verbal 
comments. 


90% 


37% 


.60 


Parent functions as her/his child's teacher. 


93% 


43% 


.72 


Parent and child interact in child-centered activities 
selected by the child. 


87% 


37% 


.53 


Parent creates an environment which contributes to 
the physical, social, and emotional well-being of 
her/his child. 


93% 


63% 


.69 



Table 4 

Classifications for the Family Literacy Portfolio 



Using the Holistic Rubric 





TOTAL 1 


ESL 


Engiish- 

speaking 


Classification 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Proficient 


3 


8.6 


0 


0 


3 


16.7 


Developing to 
Proficient 


6 


17.1 


4 


23.5 


2 


11.1 


Developing 


10 


28.6 


6 


35.3 


4 


22.2 


Emerging to 
Developing 


5 


14.3 




11.8 


3 


16.7 


Emerging 


10 


28.6 


5 


29.4 


5 


27.8 


Not Yet to 
Emerging 


1 


2.9 


0 


0 


1 


5.6 


Not Yet 


0 


0 


0 


0 


1 0 


0 
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When the classifications are examined by group, only English-speaking families 
received Proficient classifications. The percentage of classifications for the Developing 
to Proficient and Proffc/enf categories were nreariy the same for the two groups: ESL 
(23.5%) and English-speaking (27.8%). The Emerging and Emerging to Developing 
categories were also similar for the two groups: ESL (41 .2%) and English-speaking 
(44.5%). 

The results of the analytic scoring are shown in Table 5. Generally the mean 
score is three on a five-point scale. The highest mean score (Mean = 3.6 ) was for the 
creation of a positive environment and the lowest mean was for modeling reading, 
mathematics, and writirig skills (Mean = 3.2 ). Scores on the literacy goals for English- 
speaking and ESL participants were similar. The internal consistency of scores for the 
analytic rubric, as measured by Cronbach’s alpha, was .92. 



Table 5 

Scores for the Family Literacy Goals 
Using the Analytic Rubric 



• 


Total 


ESL 


English- 

Speaking 


Goals on Analytic Rubric 


Mean 


Standard 

Deviation 


Mean 


Standard 

Deviation 


Mean 


Standard 

Deviation 


Parent models reading/storyteliing, writing, 
math skills for her/his child. 


3.2 


0.9 


3.4 


0.9 


3.1 


0.9 


Parent models for his/her child goal-setting 
and planning of activities. 


3.5 


0.9 


3.6 


0.7 


3.5 


1.1 


Parent promotes self-esteem in child by 
providing, for example, physical contact 
and/or positive verbal comments. 


3.4 


0.8 1 


3.3 


0.7 


3.5 


1.0 


Parent functions as her/his child’s teacher. 


3.5 


1.0 


3.5 


0.9 


3.5 


1.1 


Parent and child interact in child-centered 
activities selected by the child. 


3.3 


0.7 


3.4 


0.7 


3.2 


0.8 


Parent creates an environment which 
contributes to the physical, social, and 
emotional well-being of her/his child. 


3.6 


0.9 


3.6 


0.8 


3.5 


0.9 



The results for the Home Screening Questionnaire are shown in Table 6. Based 
on a parent’s responses to a series of closed-response questions and a checklist of 
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toys available for the child, the HSQ provides a total score which is then used to classify 
the home environment as Normal or Suspect Two-thirds of the home environments of 
the participants were classified Suspect ESL families were classified as Suspect at 
twice the rate as the English-speaking families. 



Table 6 



Classifications for the Home Screening Questionnaire 





TOTAL 1 


1 ESL 


1 Engiish- 

1 Speaking 


Classification 


Number 


Percent 


I Number 


Percent 


1 Number 


Percent 


Normal 


7 


33.3 


1 


10.0 


1 6 


54.5 


Suspect 


14 


66.7 


9 


90.0 


.! 5 


•45.4 



As shown in Table 7 the correlation of the raw score on the HSQ and the score 
for the portfolio was r = .33. In Figure 1 the moderate, positive relationship between the 
HSQ and portfolio assessment can be seen. The portfolio classification and the HSQ 
most highly correlated (r = .42) with the goal of "creates an environment which 
contributes to the physical, social, and emotional well-being of her/his child." The 
lowest correlation (r = .02) for the HSQ and the item-level portfolio scores was for the 
goal "functions as her/his child's teacher." 



Table 7 

Correlations fcfr the Portfolio Rating 
and the Home Screening Questionnaire 



Scores 


Correlation 


Holistic Rubric 


.33 


Analytic Rubric 




Parent models reading/storytelllng, writing, math skills for her/his child. 


.21 


Parent models for his/her child goal-setting and planning of activities. 


.29 


Parent promotes self-esteem in child by providing, for example, physical contact and/or 
positive verbal comments. 


.39 


Parent functions as her/his child's teacher. 


.02 


Parent and child interact in child-centered activities selected by the child. 


.38 


Parent creates an environment which contributes to the physical, social, and emotional 
well-being of her/his child. 


.42 
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Figure 2 



Plot of Portfolio and HSQ Scores 




Program Impact 

In their responses to the survey the family educators indicated that the portfolio 
system promoted a closer alignment of their activities in the family literacy classes and 
home visits with the goals of Even Start. They also indicated that the experience 
provided them with a “better understanding of PACT.” 

Family educators also reported they found the collection of artifacts to be very 
demanding. They reported that much of the completion of the portfolio was family- 
educator driven, rather than family driven. Tension existed for the family educators 
between the creation of a personal portfolio and collection of core items for the purpose 
of evaluation. Some family educators indicated that parents were uneasy with the use 
of the portfolios for assessing the Even Start program; however, the subject of portfolios 
never arose in interviews in which families offered a wide range of likes and dislikes 
about the components of Even Start. 



Discussion 

The collaborative efforts of the Even Start staff and an evaluator resulted in a portfolio 
system for assessment of program impact. The efforts of the team included the 
identification of goals, design of the portfolio, creation of record-keeping documents, 
writing of rubrics, and determination of a scoring system. The efforts show it is possible 
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for family literacy programs to create a portfolio system to measure family growth that is 
intimately linked with their program’s mission and goals. The link between the 
assessment and the program goals, and family educator’s intimate involvement in the 
development process, clarified for them the purpose of various components of family 
literacy and promoted alignment of their activities with program goals. 

In this small-scale evaluation, a reasonable level of consistency in scoring of the 
portfolios was achieved. The scoring of portfolios by those closely involved with service 
delivery may have improved the quality of scoring as suggested by Moss and others 
(1992). Much of the information about the use of portfolios has been based on large- 
scale assessments. The consistency may be due to the level of training offered to each 
rater; the small-scale evaluation allowed direct training of raters (unlike the classroom 
teachers in Kentucky). 

To improve the consistency of scoring it will be important to create benchmarks 
that can be archived and used for several years. The archived benchmarks can be 
supplemented to provide more examples of each proficiency level, and will promote 
consistency across years as well as across raters. Archiving benchmarks creates a 
special challenge since portfolios are returned to participants at the end of the program 
year. The variety of formats that artifacts take (e.g., videos, audiotapes, crafts) makes 
duplication of materials difficult, but electronic portfolios on the market could serve to 
create benchmarks that could be archived. 

The modest correlations between the portfolio scores and the HSQ indicate the 
two assessments appear to be measuring a common trait. The absence of a 
correlation for the two assessments in the area of the parent modeling literacy skills 
may indicate the portfolio measures additional traits not measured by the HSQ. In 
terms of assessing the program, the latter point raises the question whether the HSQ is 
aligned with program emphases enough to provide a valid indicator of change. 

It also appears the portfolio assessment system may provide information that is 
less culturally-loaded than the standardized alternative. While the distribution of 
proficiency levels were similar for the portfolios, the Home Screening Questionnaire 
classified ESL families as Suspect aX twice the rate as English-speaking families. It is 
worth restating that the portfolios of ESL families were not scored by ESL instructors. 
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thus, limiting any tendency to modify judgments according to familiarity. Since family 
educators indicated the portfolio classifications reflected the levels of family literacy in 
their families, the validity of the HSQ as a measure of family literacy for ESL families 
becomes questionable. 

The validity of the HSQ as an indicator of family literacy for ESL families arose 
as a result of the collaboration between family educators and the evaluator. As a result 
of family educators sensitizing of the evaluator to the issues faced by the ESL families 
the design of the evaluation reviewed program impact for ESL and English-speaking 
families. The evaluation results were reported in disaggregated form and the 
differential impact of the HSQ became evident. 

If the portfolio assessment is to continue, however, family educators will need to 
gain additional skills in the management of portfolios and how to involve the Even- Start 
participants in the creation of the portfolios. Linkage between the literacy activities and 
the portfolio has not been established as evidenced in one family educator’s statement 
that the most frustrating aspect of the portfolio system is: “The enormous amount of 
time the portfolio system took away from my own planning time and my time to work 
with the family on the ABE component. Family Literacy component, and PACT time 
during home visits.” That family educators do not see the creation of the portfolio as a 
collaboration is reflected in the comment: “Families take responsibility, ownership - 
portfolio not completed they should be held responsible.” 

Even though the portfolio assessment is perceived as more demanding, when 
viewed from Messick’s consequential validity perspective (1989), the differential impact 
of the HSQ and the portfolio system of assessment on the classification of ESL 
households as Suspect, warrant further review to determine the appropriate method for 
assessing family literacy-at the local level or national level. If a portfolio assessment is 
to be successful, staff members will need to perceive in an integrated fashion the goals 
of the program, their instruction/activities, and the construction of the portfolio. These 
are the challenges faced by the Even Start assessment in its second year. 
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Areas of Focus for the Farnily Literacy Portfolio 

Two areas of focus were selected to guide the selection of core items for the Family 
Literacy portfolios. The two standards selected for the focus of the portfolio collection -- 
Life Skills and Parenting -- are listed below. Benchmarks are listed as headings 
underneath each standard. Listed under each benchmark are the types of 
indicators/evidence to be collected for the portfolios. 

To create a complete portfolio, an example of each indicator should be collected each 
month. By the end of the year, three to four examples, for each benchmark should be 
in each family's portfolio. The examples should reflect fyp/ca/ performance. The core 
items serve to provide a common basis for comparison; parents, children, and family 
educators are encouraged to include other materials that will present the unique story of 
a family's development. 

LIFE SKILLS: Parent models essential life skills for child. 

Literacy; Parent models reading/storytelling, math, and writing skills for her/his child. 

• Wr.ting samples 

• Frequency of reading to child 

Strategy Development: Parent models for his/her child goal-setting and planning of 
activities. 

• Goal setting activities 

• Parent planning activities 

Self-esteem: Parent promotes self-esteem in child by providing, for example, physical 
contact and/or positive verbal comments. 

• Frequency of giving encouragement 

PARENTING: Parent uses developmentally appropriate parenting practices with 

child. 

Parents as Teachers: Parent functions as her/his child's teacher. 

• Video/audio 

• List of activities 

Parent/Child Interactions: Parent and child interact in child-centered activities 
selected by the child. 

• Symbols of PACT w/ written reflections 

• Video/audio 

• Narrative summary of PACT activities 

Family Developmental Needs: Parent creates an environment which contributes to 
the physical, social, and emotional well-being of her/his child. 

• Family literacy checklist 

• Height/weight charts 
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Family Literacy Checklist 

Parent Copy 

Name of family Date _ 

Parent 



Listed below are four general areas that contribute to family literacy. Listed 
below each area are skills that are part of family literacy. Please think about how 
often you do the skills with your child and place a check under the appropriate 
column for each of the skills. Place a check under the column titled Regularly if 
you often practice the skill with your child. Place a check under the column titled 
Occasionally if you practice this skill only once in a while. Place a check under 
the column titled Not Often if you rarely practice this skill. 



Creating a learning environment 

Talks with child, describing child's actions 

Asks open-ended questions of child 

Observes child's behavior to determine abilities and needs 

Listens to child talk 

Reads to child 



Regularly^ Occasionally^ Not Ofteni^ 



Meeting phvsicai needs 

Meets child's nutritional needs 

Meets child's health needs, including exercise 

Meets child's safety needs 

Meets child's clothing and shelter needs 



Regulariy^ Occasionally^ Not Ofterr^ 



Continue to the next page 



BEST COPY A'VMIABLE 
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Providing a nurturing environment 



Regularly^ Occasionally^ Not Often^ 



Displays verbal or physical affection to child 
Provides positive encouragement to child 
Supports child's independence 
Responds positively to child's needs 
Communicates acceptance of child 



Providing guidance 

Exhibits positive modeling of desired behavior 

Sets age-appropriate limits 

Provides natural and/or logical consequences 

Offers appropriate choices 

Encourages positive behavior 

Explains rules 

Parent Comments: 



Regularly^ Occasionally^ 



1 Does skill regularly 

2 Does skill occasionally 

3 Does skill larely 2 ( 



Not Ofteri^ 



Family Literacy Checklist 

Family Educator Copy 

Name of family . Date _ 

Family educator 



Listed below are four general areas that contribute to family literacy. Listed 
below each area are skills that are indicators of family literacy. Based on your 
observations over a two or three week period, place a check under the 
appropriate column for each of the skills. Place a check under the column titled 
Regularly if the parent often demonstrates the skill with her/his child. Place a 
check under the column titled Occasionally \i the parent demonstrates this skill 
only once in a while. If you have not yet observed the parent demonstrating the 
skills, place a check under the column titled Not Observed. 



Creating a learning environment 

Talks with child, describing child's actions 

Asks open-ended questions of child 

Observes child's behavior to determine abilities and needs 

Listens to child talk 

Reads to child 



Not 

Regularty’ Occasionally^ Observecfi 



Meeting physical needs 

Meets child's nutritional needs 

Meets child's health needs, including exercise 

Meets child's safety needs 

Meets child's clothing and shelter needs 

Continue to the next page 



Not 

• Regularly^ Occasionally^ Obsen/ed? 
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Providing a nurturing environment 



Regularly^ Occasionally^ Obsen/ecP 



Displays verbal or physical affection to child 
Provides positive encouragement to child 
Supports child's independence 
Responds positively to child's needs 
Communicates acceptance of child 



Not 

, Regularly^- Occasionally^ ObservecP 

Providing guidance 

Exhibits positive modeling of desired behavior 

Sets age-appropriate limits 

Provides natural and/or logical consequences 

Offers appropriate choices 

Encourages positive behavior 

Explains rules 

Family Educator Comments:. 



2j 



’ Demonstrates skill regularly 

2 Demonstrates skill occasionally 

3 Not observed 



Family Literacy Portfolio Scoring Rubric 



Case number 

Rater 

Date 



Level 

Proficient 

Developing 

Emerging 

Not yet 

Not scorable 



Life Skills 

Parent models reading/storytelling, writing, math skills for her/his child. 
1 2 3 4 5 



Portfolio 
content reflects 
little modeling 
of reading, 
math, and 
writing skills 



Portfolio 
content reflects 
some modeling 
of reading, 
math, and 
writing skills 



Portfolio is 
characterized 
by modeling of 
reading, math, 
and writing 
skills 



No evidence 



Parent models for his/her child goal-setting and planning of activities. 



1 


2 


3 


4 


5 


Portfolio 




Portfolio 




Portfolio is No evidence 


content reflects 




content reflects 




characterized 


little modeling 




some modeling 




by modeling of 


of goal-setting 




of goal-setting 




goal-setting 


and planning 




and planning 




and planning 


of activities. 




of activities. 




of activities. 



Parenting Skills 

Parent promotes self-esteem in child by providing, for example, physical contact and/or 


verbal comments. 

1 2 


3 


4 


5 


Portfolio 


Portfolio 




Portfolio is No evidence 


content reflects 


content reflects 




characterized 


little attention 


some attention 




by a high level 


to the 


to the 




of attention to 


development 


development 




the 


of positive 


of positive 




development 


self-esteem. 


self-esteem. 




of positive 



self-esteem. 
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Parent functions as her/his child's teacher. 








1 


2 


3 


4 


5 




Portfolio 




Portfolio 




Portfolio is 


No evidence 


content shows 




content shows 




characterized 




few activities 




sonie activities 




by the parent 




with the parent 




with the parent 




typically 




teaching 




teaching 




functioning as 




her/his child. 




her/his child. 




a teacher to 












his/her child. 




Parent and child interact in child-centered activities selected by the child. 




1 


2 


3 


4 


5 


- 


Portfolio 




Portfolio 




Portfolio is 


No evidence 


content shows 




content shows 




characterized 




little parent 




some parent 




by the parent 




and child 




and child 




and child 




interaction in 




interactions in 




interactions in 




activities 




activities 




activities 




which are 




which are 




which are 




selected by the 




selected by the 




selected by the 




child. 




child. 




child. 





Parent creates an environment which contributes to the physical, social, and emotional well- 
being of her/his child. 



1 


2 


3 


4 


5 


Portfolio 




Portfolio 




Portfolio is No evidence 


content leflects 




content reflects 




characterized 


an 




an 




by an 


environment 




environment 




environment 


that 




that 




that 


contributes 




contributes 




contributes to 


little to the 




somewhat to 




the physical. 


physical. 




either the 




social, and 


social, and 




physical. 




emotional 


emotional 




social, or 




well-being of a 


well-being of a 




emotional 




child. 


child. 




well-being of a 
child. 







Use the area below to make comments about evidence for decisions on Life Skills and Parenting 
Skills. 

Comments: 
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Family Literacy Portfolio Scoring Rubric 



Proficient 

The family-literacy portfolio provides a holistic picture of a family that demonstrates a high level of essential life 
skills and parenting skills. The artifacts contained within the portfolio characterize a parent who models for her/his 
child many essential life skills: literacy, strategy development, and promotion of positive self-esteem. Entries 
demonstrate that the parent models many literacy skills in reading, math, writing, and strategy development in goal- 
setting and planning of activities for her/his child. The content of the portfolio also reflects a high level of attention 
to the development of positive self-esteem in the child. 

Also, the artifacts contained within the portfolio reflect many parenting practices that are developmentally 
appropriate for the child. Activities within the portfolio demonstrate the parent functions as her/his child's teacher. 
The content of the portfolio shows a high level of parent and child engaging in child-centered activities selected by 
the child. The entries in the portfolio characterize a parent who creates an environment which contributes to the 
physical, social, and emotional needs of the child. 

Developing 

The family-literacy portfolio provides a holistic picture of a family that is developing some essential life skills and 
parenting skills. The predominant characteristic that distinguishes a Developing Level from a Proficientfevel in 
family literacy is evidence of some of the life skills and parenting skills but not all - or most — of the skills. The 
Developing Level may also reflect a high level of attention to some aspects of family literacy while other life skills 
and parent skills are not demonstrated in the portfolio. 

The artifacts contained within the portfolio characterize a parent who models for her/his child some of the essential 
life skills: literacy, strategy development, and promotion of positive self-esteem. Entries demonstrate that the parent 
models some literacy skills in reading, math, writing, and strategy development in goal-setting and planning of 
activities for her/his child. The content of the portfolio also reflects some attention to the development of positive 
self-esteem in the child. 

The artifacts contained within the portfolio reflect some parenting practices that are developmentally appropriate for 
the child. Activities within the portfolio demonstrate the may parent function as her/his child's teacher. The content 
of the portfolio may show parent and child engaging in child-centered activities selected by the child. The entries in 
the portfolio may characterize a parent who creates an environment which contributes to the physical, social, and 
emotional needs of the child. 

Emerging 

The family-literacy portfolio provides a holistic picture of a family in which a few of the essential life skills and 
parenting skills are emergent. The predominant characteristic that distinguishes an Emerging level from a 
Developing level in family literacy is evidence of a few of the life skills and parenting skills but not many of the 
skills. The portfolio is characterized by beginning development of one or two of the life skills and parenting skills 
that describe a high level of family literacy; however, development of many of the life skills and parent skills are not 
demonstrated in the portfolio. 

The artifacts contained within the portfolio characterize a parent who attempts to model for her/his child one or two 
of the essential life skills: literacy, strategy development, and promotion of positive self-esteem. Entries demonstrate 
that the parent is beginning to model a few literacy skills in reading, math, writing, and strategy development in goal- 
setting and planning of activities for her/his child. The content of the portfolio also reflects limited attention to the 
development of a positive self-esteem in the child. 

The artifacts contained within the portfolio reflect attempts at a limited number of parenting practices that are 
developmentally appropriate for the child. Activities within the portfolio demonstrate the parent has begun to 
function as her/his child's teacher. The content of the portfolio may show parent and child beginning to engage in 
child-centered activities selected by the child. The entries in the portfolio may characterize a parent who has started 
to create an environment which contributes to the physical, social, and emotional needs of the child. 
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Not yet 

The family-literacy portfolio provides a holistic picture of a family that has not yet developed essential life skills and 
parenting skills. The predominant characteristic that distinguishes a Not-Yet Level from an Emergent Level in family 
literacy is the artifacts within the portfolio lack overall evidence of any of the life skills and parenting skills. The 
portfolio is characterized by an absence of the life skills and parenting skills that describe a high level of family 
literacy. The portfolio may also demonstrate an area in such great need of development that while the parent 
demonstrates some literacy and family skills, the area in need of development is of greater consequence. 

The artifacts contained within the portfolio characterize a parent who does not model for her/his child essential life 
skills: literacy, strategy development, and promotion of positive self-esteem. Entries demonstrate that the parent 
does not model literacy skills in reading, math, writing, and strategy development in goal-setting and planning of 
activities for her/his child. The content of the portfolio also reflects no attention to the development of positive self- 
esteem in the child. 

The artifacts contained within the portfolio reflect parenting practices that may not be developmental ly appropriate 
for the child. Activities within the portfolio demonstrate the parent does not function as her/his child's teacher. The 
content of the portfolio shows parent and child typically not interacting or primarily engaging in adult-centered 
activities selected by the parent. The entries in the portfolio characterize a parent who has not created an 
environment which contributes to the physical, social, and emotional needs of the child. 



Not scorable 

The family literacy portfolio is not scorable if the family has been in the family literacy program for less than three 
months. In addition, the portfolio is not scorable if it lacks artifacts which provide evidence of family literacy in life 
skills and parenting skills. It is possible that there are few artifacts of any type contained within the portfolio. It is 
also possible that the artifacts contained within the portfolio do not provide sufficient information to tell whether a 
family is at the Not-yet Level, the Emergent Level, Developing Level, or the Proficient Level, and, thus, the portfolio 
could not be scored. 



